Release notes

Implements current batch event count and byte size metrics.

What does this PR do?

Introduce a new gauge metric to collect list of values, used a a couple, the first element count of events in the batch and the second one is the estimated memory occupation of the batch.
This list is later grabbed in the API layer to populate the two current values for event_count and batch_size.

Why is it important/What is the impact to the user?

Exposes the current batch size in terms of events and estimated memory consumption.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Start Logstash with pipeline.batch.metrics.sampling_mode set to full running an HTTP input pipeline, load it with wrk and verify that the metric API pipelines.main.batch.event_count.current and pipelines.main.batch.byte_size.current goes down to 0 when wrk is interrupted and gets a value other than 0 when the load wrk resumes.

1. Create a file named input_sample.txt with some content
2. Create a script send_file.lua as

wrk.method = "POST"
local f = io.open("input_sample.txt", "r")
wrk.body   = f:read("*all")%

3. enable the metric in config/logstash.yml:

pipeline.batch.metrics.sampling_mode: full

4. run Logstash with pipeline:

input {
  http {
    response_headers => {"Content-Type" => "application/json"}
    ecs_compatibility => disabled
  }
}
output {
  sink {}
}

5. open a shell to verify the metrics on Logstash:

while true; do curl http://localhost:9600/_node/stats | jq .pipelines.main.batch; sleep 1; clear; done

6. start and stop wrk, checking the metrics:

wrk --threads 4 --connections 12 -d10m -s send_file.lua --latency http://localhost:8080

Related issues

• Closes Implements current value for batch's byte size and event count average metric #17997
• Relates Measure average batch byte size and event count #18000